perf(coro): compact cancellation callback dispatch state - #1055

Description

Replace the mutex and condition variable embedded in every cancellation callback node with a native-width atomic dispatch phase and C++20 wait/notify synchronization.

The state machine preserves the current cancellation contract: LIFO dispatch, immediate cancellation, same-dispatch suppression of a selected later callback, non-blocking callback reentry across cancellation sources, external teardown waiting for callback and payload destruction, first-exception propagation after all selected callbacks run, and exactly-once callback-payload destruction.

On GCC 12 x86_64, the base callback node shrinks from 208 to 112 bytes and the task-parent callback node from 224 to 128 bytes. A small-buffer callback's single requested make_shared allocation falls from 224 to 128 bytes. Pinned, interleaved Release measurements show improvements across registration, unlink, dispatch, immediate-cancel, and concurrent-unregister paths.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Performance improvement (optimization that improves speed/memory usage)
• Documentation (changes to documentation, comments, or examples)
• Refactoring (code changes that neither fix bugs nor add features)
• Tests (adding or modifying tests)
• Build/CI (changes to build system, CI configuration, or dependencies)

Related Issues

Closes #1053
Related to #1048

Changes Made

Core Changes

• Replaced per-node std::mutex, std::condition_variable, and invoking-thread state with a native-width atomic phase:
  registered -> claimed -> invoking -> completed, with terminal unregistered transitions.
• Publish an immutable dispatcher identity before the release transition to claimed; acquire transitions make it visible to dispatch and teardown.
• Preserve same-dispatch suppression by allowing only the current synchronous dispatcher to transition a selected claimed callback to unregistered.
• Preserve self/cross-source callback reentry by deferring teardown rather than waiting inside callback dispatch, preventing wait cycles.
• Make external teardown wait only when it races claimed or invoking, and publish completed only after callback invocation and payload destruction.
• Retain callback exception aggregation: each node reaches a terminal phase and destroys its payload before rethrowing to the dispatcher, which continues through later selected callbacks.
• Keep a 32-bit atomic phase so supported libstdc++ implementations can use their native wait path rather than the shared hashed waiter pool used for byte atomics.
• Added deterministic hooks and regressions for claimed/invoking teardown, payload-destruction visibility, exceptions, same/cross-source reentry, suppression, inline/heap payloads, and task-parent cancellation links.
• Added an isolated Release benchmark for node layout, requested allocation bytes, register/unregister, list unlink, dispatch, immediate cancellation, and concurrent unregister tails.

API Changes (if applicable)

No public API, layout, result, callback ordering, or exception-specification change. cancel_registration::unregister() remains noexcept.

The layout change is confined to the internal coro::detail::callback_node and its internal task-parent specialization.

Migration Guide (if breaking change)

Not applicable.

Testing

Unit Tests

• Added new tests for the changes
• Updated existing tests if needed
• All tests pass locally

Integration Tests

• Tested with existing task/cancellation integration suites
• Tested in real-world scenarios (if applicable)

Sanitizer Testing

• Tested with ASAN (AddressSanitizer)
• Tested with TSAN (ThreadSanitizer)
• No new warnings or errors

Test Results

Focused Debug:
  [cancel_token]:          3,886 assertions / 75 cases, passed
  [execution_context]:       125 assertions / 16 cases, passed
  [cancellation_context]:    576 assertions / 12 cases, passed

Full normal:
  13,406 assertions / 833 cases, passed

Full ASAN:
  13,408 assertions / 833 cases, passed
  no AddressSanitizer, LeakSanitizer, or runtime-error diagnostics

Full TSAN:
  13,403 assertions passed
  833 cases: 832 passed, 1 expected existing fork-under-TSAN skip
  no ThreadSanitizer or data-race diagnostics

All builds were out of source with explicit --parallel 2. Debug and Release benchmark targets build cleanly with warnings as errors. cancel_callback_benchmark --smoke exercises every series; invalid zero, non-numeric, and extra arguments return a usage error.

Checklist

Code Quality

• My code follows the project's code style
• I have added/updated comments for complex logic
• I have removed any debug code, TODOs, or commented-out code
• My changes generate no new warnings

Documentation

• I have updated documentation (wiki, README, code comments)
• I have added examples for new features (if applicable)
• I have updated API documentation (if applicable)

Testing

• I have added tests that prove my fix is effective or my feature works
• New and existing unit tests pass locally with my changes
• I have tested with ASAN and TSAN

Compatibility

• My changes are backward compatible (or I've documented breaking changes)
• I have considered the impact on existing users
• I have updated CHANGELOG.md (if applicable)

Performance (if applicable)

• I have considered the performance impact
• I have added benchmarks for performance-critical changes

Screenshots / Diagrams

Not applicable.

Additional Notes

The Release comparison used the same final benchmark source and GCC 12 flags for both variants; only the Elio include root differed. It ran 20 interleaved baseline/candidate process pairs. Scalar series were pinned to one CPU; concurrent unregister used two CPUs and rejected samples without both removed and invoked outcomes. Confidence intervals are paired bootstrap intervals over per-process candidate/baseline ratios.

Layout and requested allocation results:

The allocation figures are requested control-block bytes recorded around make_shared; they are not allocator usable-size or tcache-class measurements.

Selected paired p50 results:

Dispatch p99 ratios were 0.820 for 1 callback, 0.526 for 8, 0.533 for 32, and 0.554 for 256; every paired confidence interval remained on the improvement side.

Concurrent unregister / 32 callbacks improved from 3,476.5 to 2,471.5 ns at p50, 5,187.5 to 4,368 ns at p95, and 5,904 to 5,136 ns at p99. The paired p99 ratio was 0.862 [0.825, 0.943]. All 40 baseline/candidate samples contained both removal and invocation outcomes; aggregate baseline outcomes were 11,627 removed / 308,373 invoked and candidate outcomes were 10,400 / 309,600.

Reviewer Guidance

Areas requiring special attention:

• Release/acquire publication of dispatcher_identity through the claimed phase.
• Same-dispatch selected-callback suppression versus cross-source callback reentry.
• The boundary requiring callback payload destruction before publishing completed.
• Exception cleanup and continued LIFO dispatch of later selected callbacks.
• Task-parent callback removal/deactivation under concurrent parent cancellation.

Questions for reviewers:

• Can any path publish a terminal phase before the last thread that must observe payload destruction?
• Can callback reentry wait on or suppress a callback selected by a different dispatcher?
• Are all payload destruction paths exactly once for registered unlink, selected suppression, normal invocation, and throwing invocation?

Thank you for contributing to Elio! 🎉